DOCUMENT RESUME 



ED 378 558 



CS Oil 978 



AUTHOR 
TITLE 



INSTITUTION 



SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Guthrie, John T.; And Othsrs 

Systemic Reform of Literacy Education: State and 
District-Level Policy Changes in Maryland, Reading 
Research Report No. 27. 

National Reading Research Center, Athens, GA. ; 
National Reading Research Center, College Park, 
MD. 

Office of Educational Research and Improvement (ED), 

Washington, DC. 

94 

117A20007 

25p.J For a related document, see CS Oil 979. 
Reports - Research/Technical (143) 

MF01/PC01 Plus Postage. 

^Educational Change; Elementary Secondary Education; 
^Evaluation Methods; Instructional Innovation; 
Language Arts; ^Literacy; *Reading Achievement; 
Reading Ins truction; State Standards ; *Student 
Evaluation; Writing Instruction 

Maryland School Performance Assessment Program; 
*Pei f ormance Based Evaluation 



ABSTRACT 

This study examines the systemic reform process that 
was initiated in Maryland in 1991. The Maryland School Performance 
Assessment Program (MSPAP) consists of learning outcomes, a framework 
for performance assessment, guidelines for school decision making, 
and suggestions for staff development. One year following the 
initiation of the MSPAP, a semistructured interview was conducted 
with the district-level administrators responsible for reading and 
language arts in the state. They were asked about their perception of 
the Assessment objectives and about the influence of the program on 
district goals and policies for instructional change. Learning 
outcomes were accurately perceived as requiring students to apply 
language to real-world problems and to construct meaning from a 
diversity of texts. The most frequently cited instructional 
innovations were the teaching of reading and language arts through 
subject matters of the curriculum and the use of trade books for 
reading instruction. The MSPAP was perceived by 247. of administrators 
as legitimating reforms already underway in their districts. Findings 
suggest that efforts toward reform should be linked to a sustained, 
change-oriented political process informed by actual data on the 
status of the reforms. (Contains 20 references and 1 table of data.) 
(Author/RS) 
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About the National Reading Research Center 



The National Reading Research Center (NRRC) is 
funded by the Office of Educational Research and 
Improvement of the U.S. Department of Education to 
conduct research on reading and reading instruction. 
The NRRC is operated by a consortium of the Universi- 
ty of Georgia and the University of Maryland College 
Prrk in collaboration with researchers at several institu- 
tions nationwide. 

The NRRC's mission is to discover and document 
those conditions in homes, schools, and communities 
that encourage children to become skilled, enthusiastic, 
lifelong readers. NRRC researchers are committed to 
advancing the development of instructional programs 
sensitive to the cognitive, sociocultural, and motiva- 
tional factors that affect children's success in reading. 
NRRC researchers from a variety of disciplines conduct 
studies with teachers and students from widely diverse 
cultural and socioeconomic backgrounds in pre-kinder- 
garten through grade 12 classrooms. Research projects 
deal with the influence of family and family-school 
interactions on the development of literacy; the interac- 
tion of sociocultural factors and motivation to read; the 
impact of literature-based reading programs on reading 
achievement; the effects of reading strategies instruction 
on comprehension and critical thinking in literature, 
science, and history; the influence of innovative group 
participation structures on motivation and learning; the 
potential of computer technology to enhance literacy; 
and the development of methods and standards for 
alternative literacy assessments. 

The NRRC is further committed to the participation 
of teachers as full partners in its research. A better 
understanding of how teachers view the development of 
literacy, how they use knowledge from research, and 
how they approach change in the classroom is crucial to 
improving instruction. To further this understanding, 
the NRRC conducts school-based research in which 
teachers explore their own philosophical and pedagogi- 
cal orientations and trace their professional growth. 
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Abstract. This study examines the systemic reform 
process that was initiated in Maryland in 199L Hie 
Maryland School Performance Assessment Program 
(MSPAP) consists of: learning outcomes, a frame- 
work for performance assessment, guidelines for 
school decision making, and suggestions for staff 
development. One year following the initiationofthe 
MSPAP, a semistructured interview was conducted 
with the district-level administrators responsible for 
reading and language arts in the state. They were 
asked about their perception of the Assessment 
objectives and about the influence of the program on 
district goals and policies for instructional change. 
Learning outcomes were accurately perceived as 
requiring students to apply language to real-world 
problems and to construct meaning from a diversity 
of texts. The most frequently cited instructional 
innovations were the teaching of reading and lan- 
guage arts through subject matters of the curriculum 
and the use of trade books for reading instruction. 
The MSPAP was perceived by 24% of administra- 
tors as legitimating reforms already underway in 
their districts. The implications of using thj MSPAP 
approach nationally are discussed. 



Leadership in education at the national level is 
placing an increased emphasis on systemic 
reform. Although there are many perspectives 
on this issue (Fuhrman, 1993), we have used 
the framework described by Smith and O'Day 
(1991) as the point of departure for our exami- 
nation of systemic school reform. Smith and 
O'Day emphasize three components in the 
process: 

1. A unifying vision and goals that are in- 
formed by underlying values concerning 
intellectually stimulating and engaging 
education for all students and that can be 
communicated and measured in a convinc- 
ing fashion 

2. A coherent instructional guidance system 
that contains content goals, curricular 
materials, professional development op- 
portunities for teachers, and assessments 
that monitor progress toward goals and 
support superior instruction 
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3 . A restructured governance system in which 
schools are accorded autonomy and teach- 
ers participate in decision making to utilize 
resources allocated by districts 

At the heart of several different systemic 
reform plans (Ravitch, 1993; Smith & O'Day. 
1991; Spady, 1994) are statements of desirable 
student outcomes. In all cases, these concern 
how well students can show their knowledge, 
use their learning strategies, pursue their per- 
sonal interests, participate in social organi- 
zations, and contribute productively to an 
expanding technological economy. Unlike the 
past in which basic skills were emphasized, 
goals for reading are now expressed in terms of 
multiple literacies that encompass how people 
communicate, think, and address problems. 
The knowledge upon which literacies draw is 
seen as personal, not objective; contingent, not 
fixed. Literate people are engaged in the con- 
stant creation, interpretation, and reinterpre- 
tation of knowledge, weaving it in and out of 
personal and societal narratives about the most 
fundamental human concerns (Brown, 1991a). 

Such higher-order outcomes are accentu- 
ated in outcome-based education. Spady (1994) 
recommends that students should demonstrate 
transformational literacy (real-life function- 
ing), transitional literacy (complex task per- 
formance), and literacy in the traditional sense 
(discrete, simple processes). The pivotal nature 
of outcomes in the systemic reform process is 
underscored by Ravitch (1993) who notes that 
"Standards are the starting point of education 
reform .... In the absence of national stan- 
dards, we have evolved a haphazard, acciden- 
tal, disconnected curriculum based on mass- 



market textbooks and standardized, multiple- 
choice tests. Education-reform must begin with 
broad agreement on what children should 
learn" (p. 772). 

Just as standards for mathematics learning 
were devised and widely disseminated by the 
National Council of Teachers of Mathematics, 
and science standards have recently been 
formulated by the American Association for 
the Advancement of Science for science educa- 
tion, the National Council of Teachers of 
English and the Internationa* Reading Associa- 
tion are developing standards in English and 
language arts. In addition to these projects of 
professional associations, standards in English, 
mathematics, and other subjects are being 
composed by the New Standards Project, 
which is privately funded and conducted 
through the Learning Research and Develop- 
ment Center and the National Center for Edu- 
cation and the Economy. These initiatives all 
define the aim of education as one of instilling 
those qualities in students that will foster the 
best interests of citizens and the cultural and 
economic growth of society. 

A second phase in systemic reform is the 
development of an instructional guidance 
system (Smith & O'Day, 1991). In addition to 
curricular goals and materials, assessments are 
incorporated into instructional guidance; these 
art designed to measure how fully students are 
attaining outcomes and standards. Assessments 
are expected to depart from traditional stan- 
dardized, norm-referenced measures because 
the assessments are criterion-referenced, and 
their content validity is at a premium. Also t the 
tasks must be inherently interesting, valuable, 
and productive for the student, as well as sensi- 
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tive to the educational and cultural circum- 
stances of the schools. The outcomes of the 
assessments are not intended primarily to 
differentiate among students or to predict future 
performance but to define how well students 
are performing tasks that will be valuable to 
them in future schooling and in their cultural 
participation. Several investigations have 
documented the usefulness of portfolio (perfor- 
mance) assessments for those purposes (Scha- 
fer, Guthrie, Almasi, & Afflerbach, 1994; 
Valencia, Hiebert, & Afflerbach, 1993). 

The third phase of the reform scenario in- 
volves school responsibility and governance. 
Assessments themselves do not improve learn- 
ing. Policy makers and teachers concur that 
change in instruction will be the immediate 
source of improved student achievement. In the 
scenario depicted by Ravitch (1993), schools 
are assumed to be responsible institutions that 
are sensitive to the basis on which they are 
being judged. When assessments include an 
opportunity to provide visibility for school 
accomplishments and are viewed as credible by 
teachers and administrators, schools will revise 
and redesign their activities to promote the 
attainment of those goals. This emphasis on 
school responsibility was articulated in America 
2000 during the Bush administration and by 
President Clinton in his campaign publication, 
Putting People First. However, Clark and 
Astuto (1994) contend that a system emphasiz- 
ing teacher and school accountability for meet- 
ing goals v/ill undermine teachers* profession- 
alism and generate antagonism among school 
personnel. 

Enhancement of school responsibility 
should be based on the development of commu 



nities of teachers who are treated like profes- 
sionals, not on accountability procedures, 
argue Clark and Astuto (1994). As decision 
makers, teachers should participate in com- 
munities of democratic discourse (Darling- 
Hammond, 1993). Principals jhould lead the 
professional community to design, critique, 
and improve structures and norms of profes- 
sional practice and reflection. Central to pro- 
ductive school communities are organizational 
plans that create time for teacher planning and 
sharing as well as cultivate a culture of com- 
mon beliefs about the learning and teaching of 
literacy (Donahoe, 1993). At the state level, a 
variety of administrations have emphasized 
site-based management procedures that allow 
and encourage schools to exercise freedom in 
rethinking instruction for the purpose of attain- 
ing commonly held goals of student learning. 
Although some authors claim that restructuring 
the governance system is the essence of sys- 
temic reform (Sashkin & Egermeier, 1993), 
this view may not place sufficient emphasis on 
instructional revision. 

The organizations that promote and dis- 
seminate education standards do not expect that 
setting standards will accelerate student learn- 
ing. Nor do they assume that assessments 
designed to measure standards will themselves 
promote student learning. Their pivotal as- 
sumption is that schools will generate new 
instructional designs, reorganize themselves to 
implement these designs, and follow through 
on implementation to ensure that student learn- 
ing follows the profile written into the original 
statement of outcomes. 

The systemic approach to educational 
reform that is embraced by many national lead- 
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ers is precarious because it rests on the un- 
tested hypothesis that a consensus regarding 
standards and the administration of suitable 
assessments will induce creative revisions of 
instruction that modify student learning. Al- 
though the hypothesis is plausible, and it has 
been assumed to be true in some circles (New- 
ton & Tarrant, 1992), the hypothesis has not 
been tested or confirmed, as would seem ad- 
visable, considering its central role in this 
national movement. The aim of this study is to 
examine this hypothesis. 

Assumptions Regarding the Public Interest 

Some writers raise doubts about the sys- 
temic reform process (Brown, 1991b), and 
some writers contend that "we do not need new 
national standards or tests in schools" (Shan- 
non, 1993, p. 90). To clarify our stand, we 
state several fundamental assumptions regard- 
ing education in a democratic society that un- 
dergird the national school reform proposals 
and the reform initiative in Maryland. Al- 
though most readers will find these precepts to 
i>e self-evident, others will not necessarily 
agree with all of them. 

State education systems are constituted by 
public law for the public good. Citizens from 
many walks of life invest a portion of their 
income for the education of their own chil- 
dren—a personal benefit. The public further 
supports the teaching of all children for the 
benefit of the culture in which they live. The 
public interest is personal, cultural, and eco- 
nomic. Because the public assumes the respon- 
sibility for funding and sustaining schools, the 
same public claims a legitimate right to partici- 



pate in defining the aims and shaping the 
means of education. Not only does the public 
have the right to participate in education, it is 
obliged to formulate desirable outcomes for its 
youth and to set sensible plans into motion for 
attaining these outcomes. 

The population affirms, by virtue of its 
voting patterns, that it believes (1) in the im- 
portance of articulating a set of collective 
educational aims; (2) in the prudence of de- 
signing a set of efficient means to attain these 
aims; and (3) in the good sense of monitoring 
how well the aims are being accomplished. In 
a rational society, the future of children is con- 
sidered to be well-served by discussions of 
how the most widely endorsed benefits of 
schooling can be provided to the largest pro- 
portion of youth over the longest period of 
time. 

Questions Addressed in the In vestigation 

Our basic q otions were designed to help 
determine how well the reform process unfolds 
in actual practice. With regard to the newly 
instituted statewide systemic reform program, 
we asked several questions: (1) Do district- 
level administrators express understanding and 
consensus about the state-level standards for 
student outcomes? (2) Do administrators per- 
ceive the framework for performance assess- 
ment that incorporates higher-order reading 
outcomes as the authors intended? (3) What 
kind of instructional and organizational revi- 
sions are being made in this reform context? 
(4) How are instructional revisions related to 
performance assessment? 
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METHOD 

Rationale 

This study examines the policies and prac- 
tices followed during the educational reform 
process in Maryland. The rationale for choos- 
ing Maryland was L.at the blueprint for educa- 
tional reform in this state is similar to the 
designs for reform that are being promoted 
nationally. Similar to the plans presented in 
America 2000 and Putting People First, the 
Maryland School Performance Program con- 
tains the following five components: 

1 . Identification of learning outcomes (com- 
pleted in May, 1990) which include satis- 
factory performance for all students in 
reading, mathematics, science, social stud- 
ies, and writing, as well as functional 
literacy for all students 

2. Development of performance standards, or 
statements of desirable student outcomes at 
the end of designated grade levels 

3. Administration of performance assess- 
ments, which was initiated in 1990-91 

4. Implementation of instructional and school 
improvements to assist students in meeting 
the standards 

5. Publication of school, school district, and 
state-level results 



Maryland State Education Goals 

The State Board of Education adopted 
goals for public education on May 22, 1990. 
The State Department of Education developed 
ten goals that defined the state education out- 
comes at the broadest level. The goals are as 
follows: 

1 . Ninety-five percent of Maryland's students 
will start first grade ready to learn, as 
demonstrated by readiness assessments. 

2. Maryland will rank in the top five states in 
the nation on national and international 
comparisons of student achievement and 
other measures of student success. 

3. One-hundred percent of Maryland's stu- 
dents will be functionally literate in read- 
ing, writing, mathematics, and citizenship. 

4. Ninety-five percent of Maryland's students 
will achieve satisfactory levels in mathe- 
matics, science, reading, social studies, 
and writing/language arts on state-devel- 
oped measures. 

5. Fifty percent of Maryland students will 
achieve excellence in mathematics, sci- 
ence, reading, social studies, and writ- 
ing/language arts on state-developed mea- 
sures. 

6. The number of Maryland students pursu- 
ing postsecondary studies in mathematics, 
science, and technology will increase by 
50 percent. 
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7. Ninety-five percent of Maryland students 
will achieve a high school diploma and 
will be prepared for postsecondary educa- 
tion, employment, or both. 

8. Ninety percent of Maryland students v/ho 
drop cut of school will secure a high 
school diploma by age 25. 

9. One-hundred percent of Maryland citizens 
will be literate. 

10. Maryland schools will be free of drugs and 
alcohol and will provide a safe environ- 
ment conducive to learning. 

Standards in Reading 

Three aspects of reading ace contained in 
the statement of desired outcomes for students 
in reading: (1) literary experience, (2) compre- 
hending informative text, and (3) reading to 
perform tasks. The objectives in the literary 
experience category consist of the following: 

• Identify the most important event in a 
story. 

• Summarize the story, play, or poem. 

• Identify the lesson or moral in a fable. 

• Identify major conflict in a selection. 

• Identify the theme. 

• Describe the setting by drawing a pic- 
ture. 

• Complete a graphic organizer. 

• Present what main things happened. 

• Infer traits, attitudes, and motives of 
characters. 



• Interpret the expressions of characters. 

• Locate descriptions of the settings. 

• Read aloud with expression. 

• List the events that lead to the climax of 
a story. 

• Compare actions of characters to one's 
own actions in a similar situation. 

• Discuss the conflict of a selection in 
light of one's own experiences. 

• Identify feelings of characters that are 
like ones you have had. 

• Analyze the author's perspective. 

• Show how the author makes the story 
into a fantasy. 

Some of the goals that pertain to compre- 
hending informative text are as follows: 

• Summarize the passage or tell what it is 
about. 

• Restate information from the text. 

• Infer the attitudes of people in a text. 

• Describe what course of action one 
might take based on the text. 

• Tell whether you agree or disagree with 
information from a selection. 

• Identify the organizational structure of 
a selection. 

• Judge the accuracy of an account. 

• Evaluate the effectiveness of an author's 
argument. 

• Describe the significance of illustrations 
and graphics to the total selection. 

• Identify bias or propaganda in a text. 

Goals in the category of reading to per- 
form tasks include the following: 
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• Identify who wouid . use a document 
such as a map. 

• Identity how someone would use the 
document. 

• Describe the central idea in a document. 

• Paraphrase directions. 

• Fill in a form with information. 

• Draw inferences from information con- 
tained in two types of documents. 

• Identify information within a document 
that one might need to complete a task. 

• Compare a form in a document to some- 
thing in one's experience. 

• Identify points of confusion in a docu- 
ment. 

• Compare two sets of directions or two 
displays of information. 

These exemplify the goals and desired out- 
comes in the Maryland School Performance 
Assessment Program. The list is not compre- 
hensive, but it illustrates the text types and the 
response expectations. 

Construction and Administration of the 1991 
Performance Assessment 

The reading portion of the Maryland 
School Performance Assessment was construct- 
ed by reading specialists and teachers who 
represented all regions in the State of Mary- 
land. Teams of teachers designed reading and 
writing tasks based on the reading standards to 
provide students opportunities to display their 
competencies to the fullest possible extent. The 
reading materials and the tasks consisted of 
stories, long expository passages, and a variety 



of documents such as illustrations, maps, and 
directions. The questions required students to 
respond at multiple levels of comprehension 
and thinking. Lower-level reproductive tasks 
were combined with expectations for the high- 
er-order integration of ideas and critical re- 
sponse to text. Understanding concepts in 
science, social studies, and mathematics and 
presenting written expressions of those under- 
standings were required. 

The reading portion of the assessment was 
administered in conjunction with the mathemat- 
ics and writing portions during a three-day 
period in all Maryland elementary schools. 
Teachers administered assessments to students 
within their own schools, but they did not 
administer them to students for whom they had 
teaching responsibilities. Student responses 
were coded according to guidelines developed 
by teams of teachers who were led by the State 
Supervisor of Reading. The psychometric 
properties, coding schemes, student responses, 
and administrative characteristics of the assess- 
ment are the subject of another study sponsored 
by the National Reading Research Center 
(Schafer et al., 1994). It is sufficient to point 
out here that die content validity of the tasks 
was regarded as high, and the reliability of 
coding was high enough to serve the purpose 
of school-building-level instructional program 
evaluation. 

School-Based Instructional Decision Making 

Central to the design of the Maryland 
School Performance Program are instruction 
and decision making at the school level. The 
Maryland School Program is an outcome-based 
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system that holds each school accountable for 
the achievement of its students. School-based 
instructional decision making is grounded in 
the belief that those responsible for implement- 
ing a program should be actively involved in 
planning it. Faculty and administrators in each 
school should be given as much autonomy as 
possible to exercise judgment in developing the 
curriculum. The State Department of Education 
stated that "instructional decisionmaking refers 
to the process of involving school staffs in 
making decisions related to the delivery of in- 
struction designed to achieve desired student 
learning outcomes" (MSPAF, p. 2). 

It was expected that school-based improve- 
ment would be determined by the analysis and 
understanding of student achievement data. 
Using the data, schools were expected to re- 
think and possibly redesign their teaching 
strategies, grouping of students, assessment of 
student learning, instructional media, use of 
space and equipment, and parental and commu- 
nity involvement. Schools were, furthermore, 
expected to reconsider their scheduling of 
instruction, allocating of discretionary monies, 
differentiating curricula, using professional and 
paraprofessional staff, and expanding their 
external learning opportunities through field 
trips and project in the community. Although 
this decision-making framework identifies some 
possible reforms of instruction at a generic 
level, the reforms are not specific to the read- 
ing program. Our purpose was to see how poli- 
cies and practices for teaching reading were 
influenced by the Maryland School Perfor- 
mance Assessment Program. 



Purposes of the Survey 

This survey consisted of an interview with 
the administrators who were responsible for 
reading programs in each school district (coun- 
ty) in Maryland. The purpose was to describe 
these administrators 1 views of the Maryland 
Performance Assessment and the instructional 
policies and practices in reading that have been 
developed in response to the performance 
assessment program. 

Participants 

Twenty-one representatives from the 24 
school districts in Maryland participated in the 
investigation (87.5%). Nineteen were females 
and two were males. Twenty of the individuals 
worked in their respective Board of Education 
offices and held positions such as Supervisor of 
Reading (5), Supervisor of Elementary Educa- 
tion (5), Supervisor of Instruction (3), Supervi- 
sor of Language A: (3), Director of Curricu- 
lum (2), Reading Specialist, Principal and 
Coordinator of Chapter 1 and Testing. Nine of 
these individuals had doctoral degrees. Because 
school districts are organized by county in 
Maryland, the terms "school district" and 
"county" are synonymous in this report. 

Administration of the Survey 

County superintendents of instruction were 
contacted and asked to submit the names of one 
or more representatives from the county who 
were knowledgeable about the ways in which 
reading instruction in elementary schools may 
have changed as a result of the Maryland 
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School Performance Assessment Program. This 
process yielded 59 possible interviewees. One 
representative from each school district was 
contacted with a request to be interviewed; 21 
of the 24 individuals contacted participated. All 
others were invited to submit their responses to 
the interview questions in writing. 

Representatives were sent letters stating the 
purpose of the research project, explaining 
their role in the project, and describing the 
questions that would be .^sked during the inter- 
view. Interviews were conducted by telephone; 
each representative was asked the entire set of 
questions. Responses were recorded as fully as 
possible by hand. 

The interview consisted of 13 items that 
were drafted by a team of four researchers 
familiar with educational measurement, assess- 
ment, classroom procedures, and the MSPAP 
itself. The goals of the interview were to: (1) 
determine what types of changes were occur- 
ring at a county level in response to the 
MSPAP, and (2) identify one or two elementa- 
ry schools within the district that were making 
exemplary changes in response to the MSPAP. 
The second goal was included as part of an 
NRRC study of the school-level innovations 
prompted by district-level policies (Almasi, 
Afflerbach, Guthrie, & Schafer, 1994) and the 
barriers to implementing those innovations 
(Afflerbach, Almasi, Guthrie, & Schafer, 
1994). 

Survey Questions 

The questions used are presented next; 
questions 1-7 are pertinent to this study. 



1. In your view, what are the most important 
objectives measured by the reading/lan- 
guage arts portions of the Maryland 
School Performance Assessment Program 
(MSPAP)? 

2. At the time the MSPAP was first adminis- 
tered in 1991, how well-aligned were the 
reading/language arts portions of the 
MSPAP with the curricular goals of your 
county? 

3. Have your county's curricular goals 
changed as a result of the MSPAP? Can 
you describe the changes? 

4. Have there been any changes in the 
amount or type of reading and writing in- 
struction used in your county as a result of 
the reading/language arts portions of the 
MSPAP? 

5. Have there been any district-wide curricu- 
lar adaptations in terms of goals and/or 
materials used to accommodate instruction- 
al changes as a result of the MSPAF? 

6. Are students in your county better readers 
now as a result of these instructional chang- 
es? How do you know? 

7. Are there any groups of students that have 
been targeted for special activities because 
of the MSPAP? 

8 . What other instructional or curricular deci- 
sions have resulted from the MSPAP? 
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9. Can you name one or two elementary 
schools in your county that you feel repre- 
sent examples of schools that are making 
changes as a result of the MSPAP? (Ask if 
respondent thinks their county was res- 
ponding in ways similar to the MSPAP 
prior to its implementation) 

• A school that had maximum change 
irrespective of what was happening 
before the MSPAP 

• A school that had maximum change 
that is consistent with the MSPAP 

10. What innovations are being made in these 
schools? 

1 1 . Are there any special programs that char- 
acterize these schools (e.g., federal fund- 
ing, magnet schools)? How would you 
describe the relationship between adminis- 
tration and faculty at these schools? 

12. What demographic information charac- 
terizes these schools in terms of racial 
composition, attendance, socioeconomic 
status, and school population? 

• Racial composition: 

• Attendance: 

• Socioeconomic status: 

• School population: 

13. What county procedures should be at- 
tended to should we wish to interview 
teachers and other professionals in 
these schools as part of this research 
effort? 



Recording Responses to the Interviews 

Responses to each question were written in 
abbreviated form by the interviewer, who is 
the second author of this report. The first 
author compared these abbreviated responses to 
the field notes taken by the interviewer on each 
question. The abbreviated responses were 
discussed, and any differences between the two 
authors were resolved. A frequency count was 
made to determine the number of participants 
in the survey who expressed each of the differ- 
ent abbreviated responses to each of the 13 
questions in the interview. Individuals were 
permitted to give as many responses as they 
wished, so the totals often sum to more than 
2 1 . The results of the survey are presented in 
Table 1. 

RESULTS 

The findings from the survey are presented 
for each question in Table 1 . The percentage of 
reading administrators in each county that re- 
sponded to the questions is included. 

DISCUSSION 

Administrators of reading in the state of 
Maryland thought that the reading portion of 
the Maryland School Performance Assessment 
Program emphasized: (1) constructing meaning 
across a variety of texts, (2) applying language 
and critical thinking capabilities to real-world 
problems, and (3) writing extensively to ex- 
press interpretations and critical responses to 
text. The administrators viewed these objec- 
tives of the Improvement Program and the Per- 
formance Assessment as substantially different 
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Table 1. Percentage of Reading Administrators in each County that Responded to the Questions 



Question 1: In your view, what are the most important objectives measured by the reading/language arts portions of the MSPAP? 



Total 


% 


Response 


8 


38.1 


Constructing meaning across a variety of texts 


8 


38.1 


Applying language and critical thinking to real-world problems 


7 


33.3 


All are important 


7 


33.3 


Writing in response to reading stances 


5 


23.8 


Constructing meaning using different stances 


5 


23.8 


Strategic reading 


4 


19.0 


Creating motivated readers 


4 


19.0 


Integration across the curriculum 


2 


9.5 


Use of personal experiences 


2 


9.5 


Reading comprehension 


1 


4.8 


Reading as a literary experience 


1 


4.8 


Collaboration among students 



Question 2: At the time the MSPAP was first administered in 1991, how well-aligned were the reading/language arts 
portions of the MSPAP with the curricular goals of your county? 



Total % Response 

Z 23.8 Highly 

8 61.9 Moderately 

3 14.3 Not at all 



Question 3: Have your county's curricular goals changed as a result of the MSPAP? 

Total % Response 

5 23.8 Changed substantially 
8 38.1 Changed slightly 

8 38.1 No change 

Reason for No Change 

7 87.5 Had prior consistency 

1 12.5 No change yet 

Of those who reported change, how have county's goals changed? 

7 38.9 MSPAP helped refine, served as impetus, or was vehicle for implementing ideas that were 

already underway 
7 38.9 Integration across the curriculum 

6 33.3 Use of trade books, literature 
4 22.2 Developed new reading guides 

3 16.7 Emphasize the process approach more now 

3 16.7 Cooperative learning 

3 16.7 Adopted the stances (reading to perform a task, to inform, for literary experience) 

3 16.7 Developed new county-wide assessments 

2 11.1 Flexible grouping 

2 11.1 Vernacular has changed 

2 11.1 Use of strategies 

1 5.5 Provided teachers with more of an opportunity to make instructional decisions 

1 5.5 Teachers are being in-serviced 

1 5.5 Success for all students 

1 5.5 Emphasize different types of writing 
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Table 1. (continued) 



Question 4: Have there been any changes in the amount or type of reading and writing instruction used in 
your county as a result of the reading/language arts portions of the MSPAP? 



Total 


% 


Response 


13 


61.9 


Integrating reading, writing with content 


8 


38.1 


Purchased/used more trade books 


7 


33.3 


Increase in amount of writing/process writing 


5 


23.8 


Provide in-service for teachers 


4 


19.0 


Cooperative learning 


4 


19.0 


Use a language arts block 


4 


19.0 


More writing in response to reading 


3 


14.3 


► .creased use of expository text 


3 


14.3 


Read for a variety of purposes 


3 


14.3 


Write for different purposes 


3 


14.3 


Flexible grouping 


3 


14.3 


Thinking strategies/dimension of thinking 


2 


9.5 


Increased amount of reading 


1 


4.8 


Peer response 


1 


4.8 


Strategy instruction 


1 


4.8 


Emphasize positive attitudes 



Question 5: Have there been any district-wide adaptations in terms of goals and/or materials used to 
accommodate instructional changes j:s a result of the MSPAP? 



Total 


% 


Response 


10 


47.6 


Literature-bastd instruction/thematic units/integration 


6 


28.6 


Handbooks, manual, curriculum guides changed 


5 


23.8 


Developed own assessment instrument 


4 


19.0 


Dimensions of thinking 


4 


19.0 


Strategic planning committees organized 


3 


14.3 


Flexible grouping 


2 


9.5 


Grading/reporting policies changed 


2 


9.5 


Cooperative learning 


2 


9.5 


Responding to writing prompts 


2 


9.5 


None 




4.8 


Infusing staff development days into calendar 




4.8 


Resources too limited due to budget to do much 




4.8 


Notion of reading as a transaction 




4.8 


Implemented a new unit planning model for teachers 




4.8 


Collaboration with other counties 




4.8 


Supervisors work in coordination with one another 




4.8 


Developmental spelling 



from the reading tasks that are prominent in 
standardized reading measures such as the 
Comprehensive Test of Basic Skills that has 



been used previously in the state. Despite these 
trends, there was not a widespread agreement 
regarding these goals . Approximately one-third 
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Table 1. (continued) 



Question 6a: Are students in your county better readers now as a result of these instructional changes? 



Total 

C 
5 



% 
28.6 
23.8 
38.1 

9.5 



Response 

Yes 

Slightly better 

Too early to determine 

Same 



Question 6b: How do you know? 
responded "same" 



Total 

2 
6 



16.7 
50.0 



33.3 
16.7 



(N = 12) Includes all that responded "yes/' slightly better/' and one who 



Response 
Better writers 
Enjoy reading more 

• Pick up more books 

• Check out more books from school library 

• Have incentive programs 

• Observe students reading more 

• Sustained silent reading 

• Book clubs 

• County library used more 

Reading comprehension scores on CTBS improving 
County tests, basal placements tests, Gates-MacGinitie 



Question 7: Are there any groups of students that have been targeted for special activities because of the 
MSPAP? 



Total 


% 


Response 


16 


76.2 


No 


8 


50.0 


We target all children 


8 


50.0 


We target all groups but not due to MSPAP 


7 


87.5 


At-risk; Chapter 1, reading recovery 


0 


0.0 


Middle school 


1 


12.5 


Special education 


1 


12.5 


Highly able readers 


5 


23.8 


Yes 


3 


60.0 


At-risk; Chapter I, reading recovery 


2 


40.0 


Middle school 


2 


40.0 


Special education 


2 


40.0 


Highly able readers 



of the administrators thought integrated reading 
competencies were a priority; £ Jther one-third 
said critical thinking was a major goal; and 



another one-third identified writing as the most 
important outcome in the framework. These 
administrators did not possess a common 
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vision for education advocated by most writers 
on systemic reform (Fuhrman, 1993; Smith & 
O'Day, 1991). 

The administrators viewed the objectives of 
the Performance Assessment to be moderately 
well-aligned with the existing curricular goals 
of their counties. Approximately 62% of the 
administrators indicated that their goals were 
aligned moderately well with the assessment at 
the time of its first administration. Despite this 
degree of initial consistency, however, 62% 
indicated their counties' curricular goals had 
changed substantially or changed slightly to 
increase their consistency with the learning 
outcomes contained in the objectives of the 
assessment. The administrators who indicated 
that no change had occurred in the curricular 
goals stated in overwhelming numbers that 
their goals were consistent prior to administra- 
tion of the Performance Assessment and there 
was little need for altering them. 

Five types of district-level policy changes 
prevailed in these reports. The administrators 
emphasized increases in each of the following: 

• Integration of reading into other content 
areas 

• Use of trade books and literature as a 
basis for reading instruction 

• Efforts to broaden the curriculum to 
include the literacy goals 

• Cooperative learning as a more impor- 
tant feature 

• Construction and use of new county- 
wide assessments 



Because these changes brought the instructional 
goals and materials (books) into a closer align- 
ment with the desired outcomes, the policy for 
instructional guidance showed a discernable 
coherence of the kind recommended by Smith 
and O'Day (1991). 

One aspect of the systemic reform scenario 
that was not evident in the district policies for 
reading was change in the level or type of 
school governance and teacher decision-making 
(Erickson, 1990). Administrators seldom (only 
19%) formed teacher-planning teams to ad- 
dress the new student learning goals. Few 
administrators (4%) reported that they were 
working with other districts; and few (5%) 
indicated that teachers were given more in- 
structional decisions. The vibrant, school-wide 
teacher discourse advocated by Darling-Ham- 
mond (1993) and Clark and Astuto (1994) was 
not observed in these districts. 

Asked whether the students are better 
readers as a result of changes taking place fol- 
lowing the Performance Assessment Program, 
a plurality of administrators indicated that it 
was too early to talk. Those administrators 
who reported positive effects on student learn- 
ing said that they thought students were more 
fully engaged with reading as a school and 
recreational activity. They reported that stu- 
dents appeared to check out more books from 
the school library, enjoy incentive programs, 
request silent reading in schools, participate in 
book clubs, and to use the county library more 
frequently than they did prior to the adminis- 
tration of the Performance Assessment. Some 
administrators noted that they thought the 
reading comprehension scores on the compre- 
hensive test of basic skills standardized mea- 
sure were improving. Administrators reported 
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that these curricular shifts were very demand- 
ing on the time, ingenuity, and management 
capabilities of the administrators and teachers. 

The MSPAP was a catalyst for new think- 
ing among many administrators. Because the 
Assessment was designed, composed, and 
administered by teachers, it w r as reasonably 
consistent with administrators' and teachers' 
beliefs. This consistency enabled the assess- 
ment to spur the implementation of constructive 
instructional options. The Assessment led to 
the introduction of more complex reading, 
writing, and thinking activities and it spawned 
a richer context for the performance of those 
tasks than previous testing systems have per- 
mitted. The inclusion of real-world reading 
activities in the Assessment fostered the devel- 
opment of policies that encouraged instruction- 
al integrations of reading into varied contexts 
of learning, such as history, science, and litera- 
ture. However, the extent of these changes was 
not remarkable, which is consistent with expe- 
rience across the nation (Marzano, 1994). 
Although some of the districts adopted some of 
the state-wide goals, and some of the instruc- 
tional changes were tied to some of the goals, 
a broad-reaching, coherent program based on 
a vision for student literacy learning, coupled 
with an instructional guidance system and 
supported by a revised governance structure 
(Smith & O'Day, 1991), was not evident at this 
early stage of the reform process. 

Systemic reform programs should be 
judged in terms of whether students are accom- 
plishing the desired outcomes (De La Luz 
Reyes & McCollum, 1992). The question is: 
Are students reading better than they were 
before the reform initiatives? At present, we do 
not have the student achievement data to ad- 



dress this question, either in Maryland or in 
other states that have initiated outcome-based 
reforms such as Connecticut or Kentucky. But 
even if we had the data, it would be premature 
to conclude that systemic reform has succeed- 
ed, as the proponents might prefer, or that it 
has failed as the critics have argued (Clark & 
Astuto, 1994), because the fundamental condi- 
tions of reform have not been implemented. 

Many pronouncements of success and 
failure in school reform are premature. The 
Maryland experience teaches that consensus is 
not attained immediately, that the first version 
of performance assessment is not optimal 
(Schafer et al., 1994), that instructional ideas 
are often innovative (Almasi et al., 1994), but 
there are multiple barriers to fundamental 
change (Afflerbach et al., 1994). These find- 
ings suggest that efforts toward reform should 
be linked to a sustained, change-oriented politi- 
cal process (Clune, 1993), informed by actual 
data on the status of the reforms. Schools, 
districts, and states require data to inform the 
evolution of their outcomes, assessments, 
instructional practices, and governance struc- 
tures. If districts are to become learning organ- 
izations and schools are to become learning 
communities, they need to conduct reality 
checks on their own reform programs. 
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